Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow move on same object bucket during file upload. #3205

Closed
wants to merge 2 commits into from

Conversation

cfiehe
Copy link

@cfiehe cfiehe commented Sep 10, 2024

This commit fixes the issue nextcloud/server#47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.

This commit fixes the issue nextcloud/server#47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.

Signed-off-by: Christoph Fiehe <[email protected]>
@susnux susnux added enhancement 3. to review Items that need to be reviewed php Items related to PHP updates and code issues labels Sep 11, 2024
Copy link
Contributor

@artonge artonge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but would wait for @icewind1991 opinion. Maybe this logic should be moved to the server repo.

@provokateurin
Copy link
Member

provokateurin commented Sep 11, 2024

Maybe this logic should be moved to the server repo.

I'm pretty sure this is the case. It's also something I suggested in nextcloud/server#47682 as I noticed the same problem when you only upload a single (large) chunk.

@provokateurin provokateurin self-assigned this Sep 11, 2024
Agreed. Doing the comparison with the `===` operator is better, because it is stricter. Thank you very much for the suggestion.

Co-authored-by: Ferdinand Thiessen <[email protected]>
Signed-off-by: cfiehe <[email protected]>
@cfiehe
Copy link
Author

cfiehe commented Sep 11, 2024

Well, I think the fix can stay in the plugin until Nextcloud has developed further and handles file renaming on the same storage in a more efficient way than today. Unfortunately, I cannot find something similar in case of a local storage. Uploads in a group folder on a local storage always cause an inefficient copy-delete on Nextcloud triggered by the ChunkingPlugin and the general fallback in OCA\DAV\Connector\Sabre\Directory. Currently, there seems to be no way to get there in between.

@sorbaugh
Copy link
Contributor

sorbaugh commented Sep 12, 2024

Nice PR! Let's wait indeed for @icewind1991. If everything is ok, I'd say we merge this one as it would be a significant performance boost. We can then STILL move the logic to server afterwards :)

@icewind1991
Copy link
Member

This logic should probably go into ObjectStoreStorage instead to handle more cases around "cross-storage moves" that stay within the same object store.

Also instead of doing nothing/return true it should still trigger the moveFromCache to ensure that cache stays up to date.

@cfiehe
Copy link
Author

cfiehe commented Sep 14, 2024

@icewind1991 @provokateurin
I have created a new pull request nextcloud/server#47986. May someone please have a close look on it. I hope, that the change does not have any negative side effects.

@provokateurin
Copy link
Member

Closing this in favor of the server PR :)

Copy link
Contributor

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review Items that need to be reviewed enhancement feedback-requested performance 🚀 php Items related to PHP updates and code issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants